Commonly owned U.S. Pat. No. 7,123,277 entitled “CONVERSION OF A SUB-PIXEL FORMAT DATA TO ANOTHER SUB-PIXEL DATA FORMAT,” issued to Elliott et al., discloses a method of converting input image data specified in a first format of primary colors for display on a display panel substantially comprising a plurality of subpixels. The subpixels are arranged in a subpixel repeating group having a second format of primary colors that is different from the first format of the input image data. Note that in U.S. Pat. No. 7,123,277, subpixels are also referred to as “emitters.” U.S. Pat. No. 7,123,277 is hereby incorporated by reference herein for all that it teaches.
The term “primary color” refers to each of the colors that occur in the subpixel repeating group. When a subpixel repeating group is repeated across a display panel to form a device with the desired matrix resolution, the display panel is said to substantially comprise the subpixel repeating group. In this discussion, a display panel is described as “substantially” comprising a subpixel repeating group because it is understood that size and/or manufacturing factors or constraints of the display panel may result in panels in which the subpixel repeating group is incomplete at one or more of the panel edges. In addition, any display would “substantially” comprise a given subpixel repeating group when that display had a subpixel repeating group that was within a degree of symmetry, rotation and/or reflection, or any other insubstantial change, of one of the embodiments of a subpixel repeating group illustrated herein or in any one of the issued patents or patent application publications referenced below.
References to display systems or devices using more than three primary subpixel colors to form color images are referred to herein as “multi-primary” display systems. In a display panel having a subpixel repeating group that includes a white (clear) subpixel, such as illustrated in FIGS. 5A and 5B, the white subpixel represents a primary color referred to as white (W) or “clear”, and so a display system with a display panel having a subpixel repeating group including RGBW subpixels is a multi-primary display system.
By way of example, the format of the color image data values that indicate an input image may be specified as a two-dimensional array of color values specified as a red (R). green (G) and blue (B) triplet of data values. Thus, each RGB triplet specifies a color at a pixel location in the input image. The display panel of display devices of the type described in U.S. Pat. No. 7,123,277 and in other commonly-owned patent application publications referenced below, substantially comprises a plurality of a subpixel repeating group that specifies a different, or second, format in which the input image data is to be displayed. In one embodiment, the subpixel repeating group is two-dimensional (2D); that is, the subpixel repeating group comprises subpixels in at least first, second and third primary colors that are arranged in at least two rows on the display panel.
For example, display panel 20 of FIG. 2 is substantially comprised of subpixel repeating group 22. In FIG. 2 and in the other Figures that show examples of subpixel repeating groups herein, subpixels shown with vertical hatching are red, subpixels shown with diagonal hatching are green and subpixels 8 shown with horizontal hatching are blue. Subpixels that are white (or clear) are shown with no hatching, such as subpixel 6 in FIG. 5A. In FIG. 21, subpixels 1901 in subpixel repeating groups 1920 and 1923 that have a dashed-line, right-to-left diagonal hatching, indicate an unspecified fourth primary color, which may be magenta, yellow, grey, grayish-blue, pink, greenish-grey, emerald or another suitable primary. Subpixels that have a narrowly spaced horizontal hatching, such as subpixel 1902 in subpixel repeating group 1934, are the color cyan, abbreviated herein as C. Thus, subpixel repeating group 1934 shows a multiprimary RGBC repeating group. With reference again to FIG. 2, in subpixel repeating group 22, the subpixels of two of the primary colors are arranged in what is referred to as a “checkerboard pattern.” That is, a second primary color subpixel follows a first primary color in a first row of the subpixel repeating group, and a first primary color subpixel follows a second primary color in a second row of the subpixel repeating group. FIGS. 5A and 5B are also examples of a 2D subpixel repeating group having this checkerboard pattern.
Performing the operation of subpixel rendering the input image data produces a luminance value for each subpixel on the display panel such that the input image specified in the first format is displayed on the display panel comprising the second, different arrangement of primary colored subpixels in a manner that is aesthetically pleasing to a viewer of the image. As noted in U.S. Pat. No. 7,123,277, subpixel rendering operates by using the subpixels as independent pixels perceived by the luminance channel. This allows the subpixels to serve as sampled image reconstruction points as opposed to using the combined subpixels as part of a “true” (or whole) pixel. By using subpixel rendering, the spatial reconstruction of the input image is increased, and the display device is able to independently address, and provide a luminance value for, each subpixel on the display panel.
In addition, in some embodiments of the techniques disclosed in U.S. Pat. No. 7,123,277, the subpixel rendering operation may be implemented in a manner that maintains the color balance among the subpixels on the display panel by ensuring that high spatial frequency information in the luminance component of the image to be rendered does not alias with the color subpixels to introduce color errors. An arrangement of the subpixels in a subpixel repeating group might be suitable for subpixel rendering if subpixel rendering image data upon such an arrangement may provide an increase in both spatial addressability, which may lower phase error, and in the Modulation Transfer Function (MTF) high spatial frequency resolution in both horizontal and vertical axes of the display. In some embodiments of the subpixel rendering operation, the plurality of subpixels for each of the primary colors on the display panel may be collectively defined to be a primary color plane (e.g., red, green and blue color planes) and may be treated individually.
In one embodiment, the subpixel rendering operation may generally proceed as follows. The color image data values of the input image data may be treated as a two-dimensional spatial grid 10 that represents the input image signal data, as shown for example in FIG. 1. Each input image sample area 12 of the grid represents the RGB triplet of color values representing the color at that spatial location or physical area of the image. Each input image sample area 12 of the grid, which may also be referred to as an implied sample area, is further shown with a sample point 14 centered in input image sample area 12.
FIG. 2 illustrates an example of display panel 20 taken from FIG. 6 of U.S. Pat. No. 7,123,277. The display panel comprising the plurality of the subpixel repeating group 22 is assumed to have similar addressable dimensions as the input image sample grid 10 of FIG. 1, considering the use of overlapping logical pixels explained herein. The location of each primary color subpixel on display panel 20 approximates what is referred to as a reconstruction point (or resample point) used by the subpixel rendering operation to reconstruct the input image represented by spatial grid 10 of FIG. 1 on display panel 20 of FIG. 2. Each reconstruction point is centered inside its respective resample area, and so the center of each subpixel may be considered to be the resample point of the subpixel. The set of subpixels on display panel 20 for each primary color is referred to as a primary color plane, and the plurality of resample areas for one of the primary colors comprises a resample area array for that color plane. FIG. 3 (taken from FIG. 9 of U.S. Pat. No. 7,123,277) illustrates an example of resample area array 30 for the blue color plane of display panel 20, showing reconstruction (resample) points 37, roughly square shaped resample areas 38 and resample areas 39 having the shape of a rectangle.
U.S. Pat. No. 7,123,277 describes how the shape of resample area 38 may be determined in one embodiment as follows. Each reconstruction point 37 is positioned at the center of its respective subpixel (e.g., subpixel 8 of FIG. 2), and a grid of boundary lines is formed that is equidistant from the centers of the reconstruction points; the area within each boundary forms a resample area. Thus, in one embodiment, a resample area may be defined as the area closest to its associated reconstruction point, and as having boundaries defined by the set of lines equidistant from other neighboring reconstruction points. The grid that is formed by these lines creates a tiling pattern. Other embodiments of resample area shapes are possible. For example, the shapes that can be utilized in the tiling pattern can include, but are not limited to, squares, rectangles, triangles, hexagons, octagons, diamonds, staggered squares, staggered rectangles, staggered triangles, staggered diamonds, Penrose tiles, rhombuses, distorted rhombuses, and the like, and combinations comprising at least one of the foregoing shapes.
Resample area array 30 is then overlaid on input image sample grid 10 of FIG. 1, as shown in FIG. 4 (taken from FIG. 20 of U.S. Pat. No. 7,123,277.) Each resample area 38 or 39 in FIG. 3 overlays some portion of at least one input image sample area 12 on input image grid 10 (FIG. 1). So, for example, resample area 38 of FIG. 3 overlays input image sample areas 41, 42, 43 and 44. The luminance value for the subpixel represented by resample point 37 is computed using what is referred to as an “area resample function.” The luminance value for the subpixel represented by resample point 37 is a function of the ratio of the area of each input image resample area 41, 42, 43 and 44 that is overlapped by resample area 38 to the total area of resample area 38. The area resample function is represented as an image filter, with each filter kernel coefficient representing a multiplier for an input image data value of a respective input image sample area. More generally, these coefficients may also be viewed as a set of fractions for each resample area. In one embodiment, the denominators of the fractions may be construed as being a function of the resample area and the numerators as being the function of an area of each of the input sample areas that at least partially overlaps the resample area. The set of fractions thus collectively represent the image filter, which is typically stored as a matrix of coefficients. In one embodiment, the total of the coefficients is substantially equal to one. The data value for each input sample area is multiplied by its respective fraction and all products are added together to obtain a luminance value for the resample area.
The size of the matrix of coefficients that represent a filter kernel is typically related to the size and shape of the resample area for the reconstruction points and how many input image sample areas the resample area overlaps. In FIG. 4, square shaped resample area 38 overlaps four input sample areas 41, 42, 43 and 44. A 2×2 matrix of coefficients represents the four input image sample areas. It can be seen by simple inspection that each input sample area 41, 42, 43 and 44 contributes one-quarter (¼ or 0.25) of its blue data value to the final luminance value of resample point 37.
This produces what is called a 2×2 box filter for the blue color plane, which can be represented as
0.250.250.250.25In this embodiment, the area resample filter for a given primary color subpixel, then, is based on an area resample function that is integrated over the intersection of an incoming pixel area (e.g., implied sample areas 12 of FIG. 1), and normalized by the total area of the area resample function.
In the example illustrated herein, the computations assume that the resample area arrays for the three color planes are coincident with each other and with the input image sample grid 10. That is, the red, green and blue resample area arrays for a panel configured with a given subpixel repeating group are all aligned in the same position with respect to each other and with respect to the input image sample grid of input image data values. For example, in one embodiment, the primary color resample area arrays may all be coincident with each other and aligned at the upper left corner of the input image sample grid. However, it is also possible to align the resample area arrays differently, relative to each other, or relative to the input image sample grid 10. The positioning of the resample area arrays with respect to each other, or with respect to the input image sample grid, is called the phase relationship of the resample area arrays.
Because the subpixel rendering operation renders information to the display panel at the individual subpixel level, the term “logical pixel” is introduced. A logical pixel may have an approximate Gaussian intensity distribution and may overlap other logical pixels to create a full image. Each logical pixel is a collection of nearby subpixels and has a target subpixel, which may be any one of the primary color subpixels, for which an image filter will be used to produce a luminance value. Thus, each subpixel on the display panel is actually used multiple times, once as a center, or target, of a logical pixel, and additional times as the edge or component of another logical pixel. A display panel substantially comprising a subpixel layout of the type disclosed in U.S. Pat. No. 7,123,277 and using the subpixel rendering operation described therein and above achieves nearly equivalent resolution and addressability to that of a convention RGB stripe display but with half the total number of subpixels and half the number of column drivers. Logical pixels are further described in commonly owned U.S. Patent Application Publication No. 2005/0104908 entitled “COLOR DISPLAY PIXEL ARRANGEMENTS AND ADDRESSING MEANS” (U.S. patent application Ser. No. 10/047,995), which is hereby incorporated by reference herein. See also Credelle et al., “MTF of High Resolution PenTile Matrix™ Displays,” published in Eurodisplay 02 Digest, 2002, pp 1-4, which is hereby incorporated by reference herein.
Examples of three-primary color and mulit-primary color subpixel repeating groups, including RGBW subpixel repeating groups, and associated subpixel rendering operations are disclosed in the following commonly owned U.S. Patent Application Publications: (1) U.S. Patent Application Publication No. 2004/0051724 (U.S. application Ser. No. 10/243,094), entitled “FOUR COLOR ARRANGEMENTS AND EMITTERS FOR SUB-PIXEL RENDERING;” (2) U.S. Patent Application Publication No. 2003/0128179 (U.S. application Ser. No. 10/278,352), entitled “COLOR FLAT PANEL DISPLAY SUB-PIXEL ARRANGEMENTS AND LAYOUTS FOR SUB-PIXEL RENDERING WITH SPLIT BLUE SUB-PIXELS;” (3) U.S. Patent Application Publication No. 2003/0128225 (U.S. application Ser. No. 10/278,353), entitled “COLOR FLAT PANEL DISPLAY SUB-PIXEL ARRANGEMENTS AND LAYOUTS FOR SUB-PIXEL RENDERING WITH INCREASED MODULATION TRANSFER FUNCTION RESPONSE;” (4) U.S. Patent Application Publication No. 2004/0080479 (U.S. application Ser. No. 10/347,001), entitled “SUB-PIXEL ARRANGEMENTS FOR STRIPED DISPLAYS AND METHODS AND SYSTEMS FOR SUB-PIXEL RENDERING SAME;” (5) U.S. Patent Application Publication No. 2005/0225575 (U.S. application Ser. No. 10/961,506), entitled “NOVEL SUBPIXEL LAYOUTS AND ARRANGEMENTS FOR HIGH BRIGHTNESS DISPLAYS;” and (6) U.S. Patent Application Publication No. 2005/0225563 (U.S. application Ser. No. 10/821,388), entitled “SUBPIXEL RENDERING FILTERS FOR HIGH BRIGHTNESS SUBPIXEL LAYOUTS.” Each of these aforementioned Patent Application Publications is incorporated herein by reference for all that it teaches.
U.S. 2005/0225575 entitled “NOVEL SUBPIXEL LAYOUTS AND ARRANGEMENTS FOR HIGH BRIGHTNESS DISPLAYS” discloses a plurality of high brightness display panels and devices comprising subpixel repeating groups having at least one white (W) subpixel and a plurality of primary color subpixels. The primary color subpixels may comprise red, blue, green, cyan or magenta in these various embodiments. U.S. 2005/0225563 entitled “SUBPIXEL RENDERING FILTERS FOR HIGH BRIGHTNESS SUBPIXEL LAYOUTS” discloses subpixel rendering techniques for rendering source (input) image data for display on display panels substantially comprising a subpixel repeating group having a white subpixel, including, for example, an RGBW subpixel repeating group. FIGS. 5A and 5B herein, which are reproduced from FIGS. 5A and 5B of U.S. 2005/0225563, illustrate exemplary RGBW subpixel repeating groups 3 and 9 respectively, each of which may be substantially repeated across a display panel to form a high brightness display device. RGBW subpixel repeating group 9 is comprised of eight subpixels disposed in two rows of four columns, and comprises two of red subpixels 2, green subpixels 4, blue subpixels 8 and white (or clear) subpixels 6. If subpixel repeating group 9 is considered to have four quadrants of two subpixels each, then the pair of red and green subpixels are disposed in opposing quadrants, analogous to a “checkerboard” pattern. Other primary colors are also contemplated, including cyan, emerald and magenta. US 2005/0225563 notes that these color names are only “substantially” the colors described as “red”, “green”, “blue”, “cyan”, and “white”. The exact color points may be adjusted to allow for a desired white point on the display when all of the subpixels are at their brightest state.
US 2005/0225563 discloses that input image data may be processed as follows: (1) Convert conventional RGB input image data (or data having one of the other common formats such as sRGB, YCbCr, or the like) to color data values in a color gamut defined by R, G, B and W, if needed. This conversion may also produce a separate Luminance (L) color plane or color channel. (2) Perform a subpixel rendering operation on each individual color plane. (3) Use the “L” (or “Luminance”) plane to sharpen each color plane.
The subpixel rendering operation for rendering input image data that is specified in the RGB triplet format described above onto a display panel comprising an RGBW subpixel repeating group of the type shown in FIGS. 5A and 5B generally follows the area resampling principles disclosed and illustrated in U.S. Pat. No. 7,123,277 and as described above, with some modifications. In the case of a display panel such as display panel 1570 of FIG. 21 substantially comprising RGBCW subpixel repeating group 1934, the reconstruction points for the white subpixels are disposed on a square grid. That is, imaginary grid lines connecting the centers of four nearest neighbor reconstruction points for the narrow white subpixels in repeating group 1934 form a square. US 2005/0225563 discloses that for such a display panel, a unity filter may be used in one embodiment to substantially map the incoming luminance data to the white subpixels. That is, the luminance signal from one incoming conventional image pixel directly maps to the luminance signal of one white subpixel in a subpixel repeating group. In this embodiment of subpixel rendering, the white subpixels reconstruct the bulk of the non-saturated luminance signal of the input image data, and the surrounding primary color subpixels provide the color signal information.
US 2005/0225563 discloses some general information regarding performing the subpixel rendering operation for RGB subpixel repeating groups that have red and green subpixels arranged in opposing quadrants, or on a “checkerboard.” The red and green color planes may use a Difference of Gaussian (DOG) Wavelet filter followed by an Area Resample filter. The Area Resample filter removes any spatial frequencies that will cause chromatic aliasing. The DOG wavelet filter is used to sharpen the image using a cross-color component. That is to say, the red color plane is used to sharpen the green subpixel image and the green color plane is used to sharpen the red subpixel image. US 2005/0225563 discloses an exemplary embodiment of these filters as follows:
TABLE 1−0.06250−0.062500.1250−0.06250.125−0.062500.250+0.1250.50.125=0.1250.750.125−0.06250−0.062500.1250−0.06250.125−0.0625DOG Wavelet Filter+Area Resample FilterCross-Color SharpeningKernel
The blue color plane may be resampled using one of a plurality of filters, such as the 2×2 box filter shown below:
0.250.250.250.25.In the case of subpixel repeating group 1926 of FIG. 21, the blue subpixels 1903 are configured to have a narrow aspect ratio such that the combined area of two blue subpixels equals the area of one of the red or green subpixels. For that reason, these blue subpixels are sometimes referred to as “split blue subpixels,” as described in commonly-owned and copending patent application US 2003/0128179 referenced above. The blue color plane for subpixel repeating group 1926 may be resampled using the box-tent filter of (0.125, 0.25, 0.125) centered on one of the split blue subpixels.
In one embodiment for producing the color signal information in the primary color subpixels, the image date of each input pixel is mapped to two sub-pixels on the display panel. In effecting this, there are still a number of different ways to align the input image sample areas with the primary color subpixels in order to generate the area resample filters. FIG. 6 (taken from FIG. 6 of US 2005/0225563) illustrates an area resample mapping of four input image sample areas 12 to the eight subpixels of subpixel repeating group 3 shown in FIG. 5A. Input image data is again depicted as shown in FIG. 1, as an array, or grid, 10 of squares, with each square 12 representing the color data values of an input image pixel, i.e., typically an RGB triplet. FIG. 6 illustrates a portion of a resample area array for the red color plane. Subpixel repeating group 3 of FIG. 5A, shown in the dark outline in FIG. 6, is superimposed upon grid 10 in an example of an alignment in which two subpixels are substantially aligned with the color image data of one input image pixel sample area 12 on grid 10. Note that in other embodiments, one subpixel may overlay the area of several input image sample areas 12. Black dots 65 in FIG. 6 represent the centers of the red subpixels of subpixel repeating group 3 (designated as red supixel 2 in FIG. 5A). The resample area array for the red color plane comprises red resample areas such as resample areas 64 and 66 that have a diamond shape, with the center of each resample area being aligned with the center 65 of a red subpixel. It can be seen that the resample areas 64 and 66 each overlay a portion of several input image sample areas. Computing the filter coefficients for the area resample filter produces what is referred to as a “diamond” filter, an example of which is the Area Resample Filter illustrated in Table 1 above.
FIG. 7 illustrates a red resample area array 260 for a display panel configured with either subpixel repeating group 3 (FIG. 5A) or 9 (FIG. 5B), and with resample areas 64 and 66 of FIG. 6 called out. Thus, when one reproduces subpixel repeating group 3 across a larger portion of grid 10 than is shown in FIG. 6, the result is resample area array 260 of FIG. 7 for the red subpixel color plane. Note that resample area arrays for green subpixels 4, blue subpixels 8 and white subpixels 6 each may be separately considered to have a similar diagonal layout.
Other subpixel repeating groups may also give rise to primary color resample area arrays having a similar diamond shape configuration. See, for example, multi-primary six-subpixel repeating group 1936 of FIG. 21 configured as
RBGGWRwhere R, G, B and W represent red, green, blue and white subpixels, respectively. In this example, the red resample area array with reconstruction points at the centers of the red subpixels defines one diagonal arrangement of resample points and the green resample area array with reconstruction points at the centers of the green subpixels defines a similar but out-of-phase diagonal arrangement.
Note that FIG. 6 illustrates a specific alignment of subpixel repeating group 3 with input image sample grid 10 and resample area array 260 of the red color plane. US 2005/0225563 discloses that any one or more aspects of the alignment of the input image pixel grid with the subpixel repeating group, or with the resample areas for each color plane, the choice of the location of the resample points vis-à-vis the input image sample grid, and the shapes of the resample areas, may be modified. In some embodiments, such modifications may simplify the area resample filters that are produced. Several examples of such modifications are disclosed therein.
Commonly owned International Application PCT/US06/19657 entitled MULTIPRIMARY COLOR SUBPIXEL RENDERING WITH METAMERIC FILTERING discloses systems and methods of rendering input image data to multiprimary displays that utilize metamers to adjust the output color data values of the subpixels. International Application PCT/US06/19657 is published as WO International Patent Publication No. 2006/127555, which is hereby incorporated by reference herein. In a multiprimary display in which the subpixels have four or more non-coincident color primaries, there are often multiple combinations of values for the primaries that may give the same color value. That is to say, for a color with a given hue, saturation, and brightness, there may be more than one set of intensity values of the four or more primaries that may give the same color impression to a human viewer. Each such possible intensity value set is called a “metamer” for that color. Thus, a metamer on a display substantially comprising a particular multiprimary subpixel repeating group is a combination (or a set) of at least two groups of colored subpixels such that there exists signals that, when applied to each such group, yields a desired color that is perceived by the Human Vision System. Using metamers provides a degree of freedom for adjusting relative values of the colored primaries to achieve desired goal, such as improving image rendering accuracy or perception. The metamer filtering operation may be based upon input image content and may optimize subpixel data values according to many possible desired effects, thus improving the overall results of the subpixel rendering operation. The metamer filtering operation is discussed in conjunction with sharpening filters in more detail below. The reader is also referred to WO 2006/127555 for further information.
The model of exemplary subpixel rendering operations based on area resample principles that is disclosed in U.S. Pat. No. 7,123,277 and in US 2005/0225563 places a reconstruction point (or resample point) that is used by the subpixel rendering operation to reconstruct the input image in the center of its respective resample area as representing a particular subpixel's “optical-center-of-gravity.” In the discussion of exemplary subpixel rendering operations disclosed in U.S. Pat. No. 7,123,277 and in US 2005/0225563, a resample area is defined as the area closest to a given subpixel's reconstruction point (i.e., within the resample area) but not closer to any other reconstruction point in the resample area array for that primary color. This can be seen in FIG. 6 where the boundary between resample areas 64 and 66 is equidistant between the two reconstruction points 65. The extent of the area resample function is confined to the area inside the defined resample area.